Sample Efficient Deep Reinforcement Learning for Dialogue Systems with Large Action Spaces

In spoken dialogue systems, we aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. A part of this effort is the policy optimisation task, which attempts to find a policy describing how to respond to humans, in the form of a function taking the current state of the dialogue and returning the response of the system. In this paper, we investigate deep reinforcement learning approaches to solve this problem. Particular attention is given to actor-critic methods, off-policy reinforcement learning with experience replay, and various methods aimed at reducing the bias and variance of estimators. When combined, these methods result in the previously proposed ACER algorithm that gave competitive results in gaming environments. These environments however are fully observable and have a relatively small action set so in this paper we examine the application of ACER to dialogue policy optimisation. We show that this method beats the current state-of-the-art in deep learning approaches for spoken dialogue systems. This not only leads to a more sample efficient algorithm that can train faster, but also allows us to apply the algorithm in more difficult environments than before. We thus experiment with learning in a very large action space, which has two orders of magnitude more actions than previously considered. We find that ACER trains significantly faster than the current state-of-the-art.Toshiba Research Europe Ltd, Cambridge Research Laboratory - RG85875 EPSRC Research Council - RG8079

Towards end-to-end multi-domain dialogue modelling

This work was funded by a Google Faculty Re- search Award (RG91111), an EPSRC studentship (RG80792), an EPSRC grant (EP/M018946/1) and by Toshiba Research Europe Ltd, Cam- bridge Research Laboratory (RG85875

On the new very early table varieties obtained by crossing different varieties of grapevine

In order to obtain new table grape varieties which ripen before or at the same time as variety Pearl of Csaba, we performed many intervarietal crossings at our experimental station, using the following parents: Pearl of Csaba, Chasselas Bouvier, Muscat Ottonel, Queen of Vineyard, Cardinal, Dattier de Beyrouth, Ribier.The Government Commission acknowledged five very early varieties, three of which have also better agrobiological and technological characteristics than the variety Pearl of Csaba.The new variety Demir Door ripens on average 6 d, the new variety Early of Belgrade 3-5 d earlier, and the new variety Grochanka ripens at the same time or 1-3 d later as the control variety Pearl of Csaba.Due to multiple regression analysis, biological characteristics as yield, cluster and berry weights and sugar content strongly depend on meteorological conditions (temperature, rainfall and solar radiation); this is valid for all the new varieties and the variety Pearl of Csaba at all locations investigated.Analysis of variance (level of 0.05 and 0.01) shows that compared to the control variety we obtained with new varieties significantly greater yields, very significant large cluster and berry weights and very significantly better uvological characteristics of berry. The average yields for all investigated locations were increased in comparison to the control variety Pearl of Csaba: Demir Door +3.004 kg/ha, Early of Belgrade +6.259 kg/ha, Grochanka +7.065 kg/ha.Organoleptic characteristics are much improved with all new varieties which is illustrated by increased indexes.All investigated varieties are acknowledged genotypes of Vitis vinifera L, and all of these new varieties exhibited better biological and technological parameters than Pearl of Csaba

Policy committee for adaptation in multi-domain spoken dialogue systems

Moving from limited-domain dialogue systems to open domain dialogue systems raises a number of challenges. One of them is the ability of the system to utilise small amounts of data from disparate domains to build a dialogue manager policy. Previous work has focused on using data from different domains to adapt a generic policy to a specific domain. Inspired by Bayesian committee machines, this paper proposes the use of a committee of dialogue policies. The results show that such a model is particularly beneficial for adaptation in multi-domain dialogue systems. The use of this model significantly improves performance compared to a single policy baseline, as confirmed by the performed real-user trial. This is the first time a dialogue policy has been trained on multiple domains on-line in interaction with real users.The research leading to this work was funded by the EPSRC grant EP/M018946/1 ”Open Domain Statistical Spoken Dialogue Systems”.This is the author accepted manuscript. The final version is available from IEEE via http://dx.doi.org/10.1109/ASRU.2015.740487

"How May I Help You?": Modeling Twitter Customer Service Conversations Using Fine-Grained Dialogue Acts

Given the increasing popularity of customer service dialogue on Twitter, analysis of conversation data is essential to understand trends in customer and agent behavior for the purpose of automating customer service interactions. In this work, we develop a novel taxonomy of fine-grained "dialogue acts" frequently observed in customer service, showcasing acts that are more suited to the domain than the more generic existing taxonomies. Using a sequential SVM-HMM model, we model conversation flow, predicting the dialogue act of a given turn in real-time. We characterize differences between customer and agent behavior in Twitter customer service conversations, and investigate the effect of testing our system on different customer service industries. Finally, we use a data-driven approach to predict important conversation outcomes: customer satisfaction, customer frustration, and overall problem resolution. We show that the type and location of certain dialogue acts in a conversation have a significant effect on the probability of desirable and undesirable outcomes, and present actionable rules based on our findings. The patterns and rules we derive can be used as guidelines for outcome-driven automated customer service platforms.Comment: 13 pages, 6 figures, IUI 201

Characterisation of microRNAs from apple (Malus domestica 'Royal Gala') vascular tissue and phloem sap

<p>Abstract</p> <p>Background</p> <p>Plant microRNAs (miRNAs) are a class of small, non-coding RNAs that play an important role in development and environmental responses. Hundreds of plant miRNAs have been identified to date, mainly from the model species for which there are available genome sequences. The current challenge is to characterise miRNAs from plant species with agricultural and horticultural importance, to aid our understanding of important regulatory mechanisms in crop species and enable improvement of crops and rootstocks.</p> <p>Results</p> <p>Based on the knowledge that many miRNAs occur in large gene families and are highly conserved among distantly related species, we analysed expression of twenty-one miRNA sequences in different tissues of apple (<it>Malus </it>x <it>domestica </it>'Royal Gala'). We identified eighteen sequences that are expressed in at least one of the tissues tested. Some, but not all, miRNAs expressed in apple tissues including the phloem tissue were also detected in the phloem sap sample derived from the stylets of woolly apple aphids. Most of the miRNAs detected in apple phloem sap were also abundant in the phloem sap of herbaceous species. Potential targets for apple miRNAs were identified that encode putative proteins shown to be targets of corresponding miRNAs in a number of plant species. Expression patterns of potential targets were analysed and correlated with expression of corresponding miRNAs.</p> <p>Conclusions</p> <p>This study validated tissue-specific expression of apple miRNAs that target genes responsible for plant growth, development, and stress response. A subset of characterised miRNAs was also present in the apple phloem translocation stream. A comparative analysis of phloem miRNAs in herbaceous species and woody perennials will aid our understanding of non-cell autonomous roles of miRNAs in plants.</p

Evaluation of Exposure Assessment Tools under REACH: Part II-Higher Tier Tools.

Stoffenmanager®v4.5 and Advanced REACH Tool (ART) v1.5, two higher tier exposure assessment tools for use under REACH, were evaluated by determining accuracy and robustness. A total of 282 exposure measurements from 51 exposure situations (ESs) were collected and categorized by exposure category. In this study, only the results of liquids with vapor pressure (VP) &gt; 10 Pa category having a sufficient number of exposure measurements (n = 251 with 42 ESs) were utilized. In addition, the results were presented by handling/activity description and input parameters for the same exposure category. It should be noted that the performance results of Stoffenmanager and ART in this study cannot be directly compared for some ESs because ART allows a combination of up to four subtasks (and nonexposed periods) to be included, whereas the database for Stoffenmanager, separately developed under the permission of the legal owner of Stoffenmanager, permits the use of only one task to predict exposure estimates. Thus, it would be most appropriate to compare full-shift measurements against ART predictions (full shift including nonexposed periods) and task-based measurements against task-based Stoffenmanager predictions. For liquids with VP &gt; 10 Pa category, Stoffenmanager®v4.5 appeared to be reasonably accurate and robust when predicting exposures [percentage of measurements exceeding the tool's 90th percentile estimate (%M &gt; T) was 15%]. Areas that could potentially be improved include ESs involving the task of handling of liquids on large surfaces or large work pieces, allocation of high and medium VP inputs, and absence of local exhaust ventilation input. Although the ART's median predictions appeared to be reasonably accurate for liquids with VP &gt; 10 Pa, the %M &gt; T for the 90th percentile estimates was 41%, indicating that variance in exposure levels is underestimated by ART. The %M &gt; T using the estimates of the upper value of 90% confidence interval (CI) of the 90th percentile estimate (UCI90) was considerably reduced to 18% for liquids with VP &gt; 10 Pa. On the basis of this observation, users might be to consider using the upper limit value of 90% CI of the 90th percentile estimate for predicting reasonable worst case situations. Nevertheless, for some activities and input parameters, ART still shows areas to be improved. Hence, it is suggested that ART developers review the assumptions in relation to exposure variability within the tool, toward improving the tool performance in estimating percentile exposure levels. In addition, for both tools, only some handling/activity descriptions and input parameters were considered. Thus, further validation studies are still necessary

Evaluation of Exposure Assessment Tools under REACH: Part I-Tier 1 Tools.

Tier 1 occupational exposure assessment tools recommended for use under the Registration, Evaluation, Authorization, and restriction of CHemicals (REACH) were evaluated using newly collected measurement data. Evaluated tools included the ECETOC TRAv2 and TRAv3, MEASEv1.02.01, and EMKG-EXPO-TOOL. Fifty-three exposure situations (ESs) based on tasks/chemicals were developed from National Institute for Occupational Safety and Health field surveys. During the field surveys, high quality contextual information required for evaluating the tools was also collected. For each ES, applicable tools were then used to generate exposure estimates using a consensus approach. Among 53 ESs, only those related to an exposure category of liquids with vapor pressure (VP) &gt; 10 Pa had sufficient numbers of exposure measurements (42 ESs with n = 251 for TRAv2 and TRAv3 and 40 ESs with n = 243 for EMKG-EXPO-TOOL) to be considered in detail. The results for other exposure categories (aqueous solutions, liquids with VP ≤ 10 Pa, metal processing, powders, and solid objects) had insufficient measurement to allow detailed analyses (results listed in the Supplementary File). Overall, EMKG-EXPO-TOOL generated more conservative results than TRAv2 and TRAv3 for liquids with high VP. This finding is at least partly due to the fact that the EMKG-EXPO-TOOL only considers pure substances and not mixtures of chemical agents. For 34 out of 40 ESs available for chemicals with VP &gt; 10 Pa, the liquid was a mixture rather than a pure substance. TRAv3 was less conservative than TRAv2, probably due to additional refinement of some input parameters. The percentages of exposure measurement results exceeding the corresponding tool estimates for liquids with VP &gt; 10 Pa by process category and by input parameters were always higher for TRAv3 compared to those for TRAv2. Although the conclusions of this study are limited to liquids with VP &gt; 10 Pa and few process categories, this study utilized the most transparent contextual information compared to previous studies, reducing uncertainty from assumptions for unknown input parameters. A further validation is recommended by collecting sufficient exposure data covering other exposure categories and all process categories under REACH